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[57] ABSTRACT 

A manual entry interactive paper and electronic document 
handling and process system uses a pen-like instrument (PI) 
with a writing point for making written entries upon a 
physical document and sensing the three-dimensional forces 
exerted on the writing tip as well as the motion associated 
with the act of writing. The PI is also equipped with a CCD 
array for reading pre-printed bar codes used for identifying 
document pages and other application defined areas on the 
page, as well as for providing optical character recognition 
data. A communication link between the PI and an associ- 
ated base unit transfers the transducer data from the PI. The 
base unit includes a programmable processor, a display, and 
a communication link receiver. The processor includes pro- 
grams for written character and word recognition, memory 
for storage of an electronic version of the physical document 
and any hand-written additions to the document. The display 
unit displays the corresponding electronic version of the 
physical document on a CRT or LCD as a means of feedback 
to the user and for use by authorized electronic agents. 

21 Claims, 12 Drawing Sheets 
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MANUAL ENTRY INTERACTIVE PAPER 
AND ELECTRONIC DOCUMENT HANDLING 
AND PROCESSING SYSTEM 

FIELD OF THE INVENTION 5 

The invention relates to the field of handling paper 
documents and their corresponding electronic versions by 
allowing the user to interact with both versions of the same 
document simultaneously. Q 

BACKGROUND OF THE INVENTION 

Because people prefer paper documents for some tasks 
such as reading or annotating documents while electronic 
documents seem to be preferred for tasks such as storing, 15 
searching or communicating, it is desirable to provide docu- 
ment systems that maximize the user-friendly aspects of 
both approaches. 

Unfortunately, there has been a large technological barrier 
between the paper and electronic documentation worlds. For 20 
example, electronic symbols may be readily converted into 
a printed page, but printed documents are not readily trans- 
lated into compact electronic symbols. Also, the human 
interface with each of the mediums are radically different: 
writing on paper (or similar surfaces) comes naturally to 25 
children, while creating electronic documents generally 
requires the use of application software, keyboard, mouse, 
display, etc. 

The present invention provides a human interface input/ 3Q 
output device that is: 

(a) small and portable, 

(b) natural to use as a pen, 

(c) works with both electronic and paper documents, and 

(d) avoids complications and inconveniences of devices 35 
such as bulky scanners and digital pads. 

As a result, users can readily create or manipulate paper 
documents while simultaneously making corresponding 
electronically-coded images of the same created or manipu- 
lated paper document. This is accomplished by means of a 40 
hand-held, pen-like instrument that can write on paper 
documents, simultaneously sense the writing, and interpret 
and enter the written markings in the electronically-coded 
document. Also, image sensing allows the system to identify 
the type of document being subjected to manually-written 45 
modifications. 

SUMMARY OF THE INVENTION 

A manual entry interactive handling system uses a writing 50 
instrument to manually write data on an ordinary document 
(that can have pre-printed information) and automatically 
interpret the written data so that a corresponding electronic 
document containing any pre-printed document information 
is modified by adding the written data to the stored elec- 55 
tronic document. Both pre-printed data and the interpreted 
written data can be electronically displayed, the system 
comprises: 

(a) a physical document page with a surface suitable to be 
written upon that can also contain pre-printed docu- go 
ment information; 

(b) a pen-instrument for writing on the physical document 
surface, the pen-instrument having suitable transducers 
for sensing both the written data and the multi- 
dimensional physical forces (e.g. up to three rectilinear 65 
and two rotational components) exerted on the pen 
instrument when writing and for reading pre-printed 



2 

digital codes, and a communication link to transmit the 
transducer data to a base unit; and 
(c) a base unit for receiving the transducer data and 
converting the transducer data to binary digital infor- 
mation suitable for processing in the base unit CPU, or 
another computer, for document identification and for 
incorporating the written information with the pre- 
printed document information. 
In another embodiment, the transducers are capable of 
defining pen motion resulting firom movement of the pen- 
instrument for other than writing such as pointing for the 
control of an object (such as lights). 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example 
and not limitation in the figures of the accompanying 
drawings, in which like references indicate similar elements 
and in which: 

FIG. 1 shows the format of a daily CB page. 

FIG. 2 is a flow diagram of the basic operation of a CB 
system. 

FIG. 3 is a flow diagram for procedure 600 for interpret- 
ing the pen-instrument input data. 

FIG. 4 is a flow diagram of the entry select procedure 700. 

FIG. 5 shows a CB system's major components. 

FIG. 6 shows the architecture of a pen-instrument. 

FIG. 7 shows the geometry of an infrared PI positioning 
system. 

FIG. 8 shows an arrangement of strain gauge transducers 
in a pen-instrument. 

FIG, 9 shows the interconnections of the strain gauges on 
the flexible diaphragm. 

FIG. 10 is a triple bridge circuit for processing the strain 
gauge transducer data and for producing x, y, and z force 
components. 

FIG. 11 shows a PI unit block diagram. 

FIG. 12 shows a CB base unit block diagram. 

DETAILED DESCRIPTION 

A. Application Areas 

An interactive document system based on a pen-like 
instrument, or pen-instrument for convenience, has applica- 
tion in numerous document systems that include, but are not 
limited to, the following systems. 

Calendar Book: appointments entered on a physical page 
of a calendar book are sensed, interpreted, and recorded 
on an electronic calendar as well. 
Forms Processing: filling out of a prescribed form would 
automatically cause the generation of a corresponding 
electronic form, the paper version being kept, as 
needed, for legal or safety reasons. A significant cost 
reduction in computer data entry results. 
Signature Verification: improved reliability of signature 
recognition obtained by storing the signature related 
time-course by using spatial location and pressure to 
characterize the signature. These parameters are not 
available to the forger from viewing the signature. 
Interactive Fax: a facsimile system that permits users to 
receive documents by means of "links" similar to 
WebFax. This would permit the user to forward pages 
to selected users by simply writing the desired 
destination, which when sensed and interpreted would 
provide the communication "link". Also, the user can 



12/05/2003, EAST Version: 1.4.1 



6,01 

3 

annotate the document and the transmit or store the 
document with the annotation without the use of a 
facsimile scanner. 

Document Editing: any paper document can be edited by 
crossing-out words, and by the use of typographic 
notation for other editing operations, or annotated, or 
sent elsewhere. The paper modifications immediately 
affect the electronic version so that the electronic 
version can serve as the true source document because 
any access of, or communications of, that document 
would reflect all edits or annotations. 

Information Management: scanning the description of a 
reference document can have that paper (or its 
reference) stored in a designated file without photo- 
copying. 

Document Retrieval: searching of electronic databases by 
writing or circling a few keywords, thumbnail 
descriptions, or by drawing pictures, which, when 
interpreted, provide the data location. 
Note Book: using the pen-instrument for writing notes on 
paper and for electronically retrieving notes previously 
written on paper and stored from a sequentially ordered 
(and/or an otherwise organized) electronic memory file. 
B. System Details 
B.l Calendar Book (CB) 

Because of the variety of possible document systems, as 
outlined in the preceding section, the details of the invention 
will be presented using a single application from which the 
extension to other applications will be readily understood by 
those practicing the art. For this reason, the Calendar Book 
is described below. 

The Calendar Book, or "CB" hereinafter, looks like a 
conventional calendar having a page representing a segment 
of calendar time: a day, a week, a month, or a year. Each 
segment of time is further subdivided as needed: a year into 
months and weeks, a month into weeks and days, a day into 
hours and fractions thereof. Digitally coded information is 
printed on each page so that the pen-instrument can be used 
to read the pertinent digital code that identifies the page 
format and the segments of calendar time spanned by the CB 
page. 

FIG. 1 shows a page 20 from a CB using a one day per 
page format. This version allows selectable information 
options to be printed for each day. An area is set aside for a 
bar code identifier 13 for specifying the date (year, month, 
and day) and user I.D. The date (year optional) is also 
printed in area 21. Column of rectangles 22 and 22' are 
available for selecting and identifying the time by entering 
a suitable mark (such as a check mark) in the appropriate 
rectangle. Column 22 and 22' have identical meanings and 
are provided so that two entries, only 15 minutes apart may 
be entered each using a half width and indicated as separate 
by dividing line 23. For example, an appointment with 
Baker is indicated for 10:00 a.m. while an appointment with 
Charlie is indicated for 10:15 a.m. In other words, each hour 
is divided into 15 minute-segments in this example. Other 
increments such as one-half hour and 10 minutes could also 
be specified. The user's choice would only be constrained by 
the application being served and the page size (or detail) 
available. The time of day is also indicated by the position 
of a written entry for any given calendar page format. The 
pen -instrument provides the writing point and the optical, 
the positional, and the writing point to paper pressure data 
to the computer for interpretation, storage and graphical 
display. 

The graphical display can provide immediate feedback by 
use of an optional cathode ray tube (CRT) display. The 
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feedback can be a simple recreation of the written message 
by using pen-instrument positional writing point pressure 
data or the interpreted written message from a cursive 
character recognition device using the same data. In the 

5 latter case, incorrectly recognized written data can be 
scratched-out or over-written. 

Bar codes are read by a bar code scanner on the side of the 
pen-instrument. The bar code recognition device may either 
constantly scan in search of a legitimate bar code or operate 

ao when the positional sensors indicate that the pen-instrument 
is in a horizontal position suitable for reading a bar code. 
B.2 Operating Mode 

FIG. 2 is a flow diagram describing the basic operating 
mode of a particular CB. The operation begins at step 500 

15 where it is assumed that a fixed format has been prescribed 
and its description is stored with the configuration informa- 
tion for that CB. Configuration data includes calendar type, 
chronology layout defining time intervals, segments, and 
writing spaces. Also, a user list is stored against which the 

20 bar-code I.D. can be checked for validity. 

At step 501, the computer checks if the pen-instrument is 
moving horizontally, indicating that a CB operation is about 
to be initiated by reading the bar-code located in area 13. If 
not, the system goes into a wait loop in which the pen- 

25 instrument motion is monitored. If the appropriate horizon- 
tal motion is detected, the process proceeds to step 502 in 
which the bar-code I.D. is read by scanning bar-code area 13 
with the pen-instrument. If a valid bar code signal results, 
step 503 moves the process to step 505, otherwise the 

30 process goes to step 504 where an audible and/or visual 
alarm is set for informing the user that access is denied and 
the system returns to step 501. The bar-code can also be 
compared with a list of authorized codes and if a match 
exists the CB access request is validated and proceeds to step 

35 505. At this juncture, the pen-instrument user may try to read 
the bar-code again or investigate the system for the cause of 
the misread bar-code data. 

At step 505, the electronic version of the CB page can be 
displayed on a screen (CRT or LCD) with the display 

40 including all prior entries on that page. The process enters 
another wait loop through test step 506 which checks if the 
writing point pressure signal exceeds threshold. If it does, 
this is taken as an indication that a pen-instrument operation 
is initiated and the process moves to step 511 where the 

45 previous wait loop index (step 507) is cleared. Otherwise, 
the process goes into a wait loop comprising step 507 that 
counts the amount of passes (or time) that the loop is 
traversed. Step 508 tests if a preset time has been exceeded, 
and if not goes back to step 506. Otherwise, the loop count 

50 index of step 507 is cleared in step 509 and the display is 
aborted in step 510 before returning to step 501. 

Step 512 establishes the location of the written entry by 
using the gyroscopic rate information. By knowing the 
location of the bar-code page identification and by integrat- 

55 ing gyroscopic velocity information, the location of place at 
which pen pressure is exerted in step 506. This information 
is used to locate the corresponding electronic entry on the 
electronic graphical display. 

From step 512, the process moves to step 600, a procedure 

60 for interpreting the message that the pen-instrument has 
entered on the CB page. The procedure is shown in greater 
detail in FIG. 3. The multi-sensor output signals from the 
pen-instrument are in electrical form, preferably as binary 
coded data. In this example, four data types are used and 

65 processed: pen-instrument pressure processed in processor 
step 601, gyroscopic restoring forces in processor step 602, 
optical data representative of the pen-instrument written 
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message in processor step 603, and an auxiliary data signal At step 712 a test is made to determine if all message units 

representative of special cues for identifying calendar and have been displayed and, if not, the process returns to step 

position data from the printed CB page in processor step 702. 

604. Each processor step 601-604 extracts signal features If the user indicates at test step 704 that the trial candidate 

from its associated data source. A particular written data S message unit is not correct, the process goes to step 707 

entry may include a number of distinct message unit such as where the wait loop index is cleared and then to step 708 

a time code derived from the printed CB page, and a written where lhe P^^lc candidate index I, is incremented. At 

portion that may include a name and place. The message s,e P 709 > '* is check f d 10 determine if all of the candidates 

input sequence is parsed into message segments in vector have been exhausted and, if not, the process returns to step 

processor step 605 using the various processed sensor data 10 702 If ^ candidates have been exhausted the process 

from processors 601-604. Thus, the presence of the writing returns t0 the beginning via input B of FIG. 2. 

point-to-paper pressure over an interval of time indicates , [f a11 messa ge ™« are accounted for in step 712 

writing a message segment. Gaps between pressure intervals (implying the graphical message on the screen is correct), 

may indicate the gap between segments. A gap can also & e P rocess checks ,0 ^ * confllcl exists because ^ 

indicate a crossing of a "t" or the dotting of an "i" after the 15 events arc scheduled for the same time. If no conflict exists, 

word is written. In the latter case, gyroscopic data can s,e P 714 stores ^ 08 entry and the process proceeds to 

confirm the backward pen-instrument motion and subse- ^ B of FIG - 3 - K a confllct eMSts > a 0011(1101 alann ls 

quent pen-instrument pressure coupled with either a station- 8 iven on ^ display or by any other means such as 

ary dotting action or a forward sweeping crossing of a "t". ™ (**P 7 * 5 ) m t 1 ^^ 1 * f ven tBnc to 

By using time-synchronized multi-sensor data, the pen- 20 respond by the wait loop (steps 717, 718). If me response is 

instrument written message can be used to parse the input not timely, the process aborts by returning to input B of FIG. 

message into distinct and intertable acts. Parsing vector 2 ' If user 1D0 > cates that ^ nfsw messa S e should . not *» 

processor 605 produces a set of vectors describing each entered step 716 aborts the process by returning to input B 

written segment which are then identified by comparing the of nG - 2- If the user indicates at step 716 that the new entry 

parsed vectors with a stored set of exemplar in exemplar 25 sh ° uld «» entered step 719 asks if the previous message is 

store unit 607 by comparator step 606. In step 608, each to be deleted and the new entry entered. If the user indicates 

parsed segment generates a set of candidate vectors corre- bv "> Muriate signal from the pen-instrument that pre- 

sponding to a plausible parsed message segment, which are vious me ssage is to be overridden, the process goes to step 

preferably ordered in descending order of plausibility, i.e., of 720 otherwise both forms of the written message are 

being the correct message segment. 30 retained in step 721. 

The plausible candidate message units from message Process 700 of FIG. 4 is designed to select correctly 
selection step 608 arc arranged by sequence index, identified candidates from a written character recognition 
0Sk§K-l, indicating the order of occurrence of K message device instep 704. However, an optional embodiment would 
segments. The k" 1 message unit has L t plausible candidates P rovide for sending the processed data outputs from pro- 
designated by index l k> where 0SkgL t -l and l k indicates 35 cessmg steps 601-604 of FIG. 3 to an optional write 
decreasing plausibility with increasing value of J*. simulation step 609 where the processed sensor data is used 
• Because the system should provide a means for displaying ,0 generate a rephca of the user s written message without 
a facsimile reconstruction of the written message, the output ™V w / itin 8 recognition processing steps and is then da- 
of parsing vector processor 605 also provides an output to P la y cd » a copy of the hand written message on the 
locus reconstruction generator 609 that operates on the 40 document. Also, both forms of written messages (an ASCII 
gyroscopic and pen pressure data to reconstruct the locus of 6ncoded interpretation of the message and a rephca of the 
the pen point while writing. The digitized locus is passed on on i inal ) can be retained for future reference and selectively 
to entry point E of FIG. 4 for conflict resolution. In some displayed on the CB display. By providing this option the 
applications, a facsimile of the original written message is wtiiia & recognition process does not have to be 100% 
preferred and in others it provides a back-up to an alpha- 45 accurate. 

numeric interpreted representation based on character pat- I" manner, the electronically stored version of the CB 

tern recognition techniques. can served 38 ,he true document for which the paper 

Step 700 is a multi-step process as shown in FIG. 4. At version and 'rail exists in close accord with the electronic 

step 701, the message unit index, k, and the plausible version - If the paper version begins to become unmanage- 

candidate index, K, are cleared (k=0, I t =0). At step 702 the 50 able because of numerous changes, the user can obtain a new 

I t candidate of the k* message unit is retrieved from P™?" 01 " of the page which may be a new page without any 

memory and displayed graphically as trial message unit entries or a new page with all up-to-date entries, 

super-imposed on the appropriate CB page at the location Because the source exists in electronic form, electronic 

corresponding to that written on the actual page calendar by agents can use this information to aid the user by, for 

the pen-instrument in step 703. At this point in time, the CB 55 example, reminding users of appointments, performing 

user is allowed to examine the trial message unit in test step g rou P scheduling with other agents, reviewing the series 

704. Three exits from step 704 are provided: a wait loop schedule, and providing support information for a given 

(similar to that previously described) comprising wait loop a P ?°i?' m ™ t ' c 

indexing step 705 and test step 706 that examines the wait B.3 The CB System 

loop index to determine if the process should be aborted by 60 ™' CB svstem 90 - shown 111 F,G ' S > includes four ma J or 

returning to input B of FIG. 2 or returning to step 704. If the hardware components: 

user indicates by the pen-instrument that the candidate trial C 1 ) a 08 P a 8 e 10 or 20, as previously described; 

message unit is correct by making an appropriate pen- (2) a pen-instrument (PI) 91 for providing a user with a 

instrument movement (such as simulating or actually writing means for entering data (both printed digital and 

a check mark if correct and an "X" make if not), the process 65 written); 

moves to step 710 where the wait loop index is reset and theD (3) a base-unit 92 for electronically processing and storing 

step 711 where index k is incremented and l k is set to zero. data received from the PI and including processing and 
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conditioning electronics 96 and optional graphic screen Because the strain gauges are paired and mounted on 

display 95; and opposite sides of diagram 112, each pair, (121, 122), (123, 

(4) a communication channel 93 between the PI and 124), (125, 126), and (127, 128), tends to have opposite 

base-unit. forces (tension vs. compression) applied to the upper and 

B.3.1. Pen-instrument (PI) 91 5 lower strain gauges. When a force is applied in the x-y plane 

PI 91 is configured as a "normal" pen, i.e. a writing ( CB P a g e P lane ) 10 ball-point 116, diametrically opposite 

instrument, that is to be conveniently held and used for P airs tcnd to have opposite tension and compression forces, 

writing on a CB page 10 as shown in FIG. 6. In addition to However, when a force perpendicular to the x-y plane is 

providing a writing point that deposits ink (or other writing applied, all pairs tend to experience similar distortions, 

medium) on to paper surface by contact, the PI incorporates 10 These characteristics permit the separation of the ball point 

a battery power source 250, and transducers for providing force t0 be resolved into three orthogonal components (X, Y, 

information for the interpretation of the user written data or Z) where X and Y are in the x-y plane and Z is perpendicular 

printed digital information from the DB page, including the to the x-y plane. Thus, by connecting the strain gauges 

following. 121-128 as shown in FIG, 9 and applying the strain gauge 

Position Sensor 210: an inertial reference system is pro- 15 connections to the multiple bridge circuit shown in FIG. 10, 

vided by a gyroscope (gyro) in which a rotating member will the ball-point force is resolved at output terminals 152, 174, 

continue to turn about a fixed spin axis as long as no forces a nd 160 into the X, Y, and Z components, respectively, 

are exerted on the member and the member is not acceler- ^ strain S au S e brid S c circuitry of FIG. 11 is preferably 

ated. Gyro attitude transducers (free gyros) are used as mounted within the PI unit for greater noise immunity, 

two-degree-of-freedom gyros that can provide an output for 20 Although FIG. 8 shows shaft 114 parallel to the z 1 axis and 

each of two of a Pi's three attitude planes: pitch, yaw, and perpendicular to the x'-y' plane, the writing surface of a CB 

roll (or x, y, and z axes). Rate gyros are attitude-rate page (x-y plane) generally would not be perpendicular to the 

transducers that provide an output signal proportioned to z' a *is because a "naturally" held pen-instrument would 

angular velocity (time rate of change of attitude). The rate of from an included angle between shaft 114 and the CB page 

change of the two axes representing the plane of the CB page 25 surface of less than 90°. The transformation between two 

are used to interpret the motion and relative position asso- coordinate systems (x, y, z) and (x', y', z') that share a 

ciated the use of the PI. Accelerometers may also be used for common origin (the pen tip) is readily accomplished by the 

position sensors by double integration of the acceleration following vector relationship, 
signals. 

When the PI is equipped with an infrared (IR) detector, a 30 \x 
multiplicity of two or more external IR radiating scanning y 
sources located at prescribed positions can be used to track [ z 
the position of the PI by using triangulation techniques that 
measure the angle at which each IR scanning source illu- 
minates the PI . Alternatively, the PI can be equipped with an 35 where [X„, ^ y n ], for l^n^3, represents the direction 
IR transmitter and two or more external scanning receivers cosine vector of the (x, y, z) unit vector with respect to the 
located at prescribed positions can be used to track the (x\ y 1 , z 1 ) unit vectors (for example see Korn and Kom, 
position of the PI by triangulation. FIG. 7 shows the geom- "Mathematical Handbook for Scientists and Engineers," 
etry of an IR tracking system using three external positions McGraw-Hill Book Co., New York, 1961, section 3.1-12, 
labeled 501, 511, and 521. Each position has a directive 40 "Translation and Rotation of Rectangular Cartesian Coor- 
scanning beam (receiving or transmitting) 502, 512, and 522 dinate System"). 

associated respectively with positions 501, 511, and 521. By knowing the angular orientation of shaft 114 with 

The shaded area, labeled 530, represents the intersection of respect to the CB page (x-y plane) and the (x\ y\ z') 

the beams and corresponds to the estimated position of the components of force, the horizontal and vertical forces in the 

PI. 45 (x, y, z) system are readily obtained. Also, any rotation of the 

Pressure Sensors 100: strain gauges (or other pressure PI coordinates with respect to its inertial (gyro) system may 

sensitive transducers such as piezoelectric or reductive force also be accommodated by use of the above transformation, 

transducers) connected to the shaft of the pen tip of the PI The angular rotation of shaft 114 would tend to be the 

are used to record the forces applied to the tip by measuring same each time the PI is used by a given user. This would 

the force exerted on four pairs of orthogonal strain gauges 50 permit system to "learn" the proper value of the direct 

mounted on a flexible diaphragm that is attached to the inner cosines or, equivalently, the angles of rotation by taking 

walls of the PI with the pen shaft connected to and passing samples of vertical forces (z), horizontal plus vertical forces 

through at right angles to the diaphragm center. FIG. 8 (x, z), and longitudinal plus vertical forces (y, z) when a 

shows how the pressure sensors are mounted in the PI. The given user respectively exerts a vertical force on the page, 

operation of such a structure has been described by Crane et 55 writes a horizontal line, and writes a longitudinal line, 

al. in U.S. Pat. No. Reg. 29,765. The transformation of coordinates for pen tip forces is not 

FIG. 8 shows a PI cylindrical lower end 110 that houses essential because the information content of a vector is not 

a ball -point 116 on the lower end of shaft 114 containing the changed by a linear coordinate transformation. However, it 

ink reservoir. The shaft, supported by a non-conducting may be useful for the normalization of forces so that they are 

flexible diaphragm 112 that is attached to cylindrical wall 60 independent of how the pen-instrument was held. However, 

110, causes the diaphragm to flex when pressure is applied un-normalized forces may be useful for signature 

to ball-point 116. As a result, strain gauges 121-128 that are verification, because of the added parameters relating to how 

affixed to diaphragm 112 are strained (or compressed) in the pen-instrument is held and used, 

accordance with the flexing of diaphragm 112 in response to Pressure sensor information is useful for recognizing 

the forces experienced by ball-point 116. In this manner, the 65 when a written entry is made on the CB page and for parsing 

forces experienced by ball -point 116 are converted into the entry into message units for easier message recognition, 

resistance changes in the strain gauges 121-128. It is also useful for signature verification. 
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Optical Sensor 220: a charge-coupled-device (CCD) opti- It should be noted that had any of the other applications 

cal scanning transducer is used for reading reflected light listed in section A of the Detailed Description, the basic 

variations from a scanned portion of the CB page covered by system configuration would be the same: 

the CCD array. The transducer is used to scan the identifying (a) a physical document with a writing surface and 

CB page bar-code. The CCD array can be located on the PI $ pre-printed information; 

in several different areas: ( D ) a pen-instrument for reading pre-printed (bar code) 

(a) a linear side mounted array 220 along the length of the information, for sensing writing forces, for sensing 
PI is useful for scanning a large segment of the page; pen-instrument motion, for sensing written characters, 

(b) a CCD array 260 at the tip of the PI pen or to one side and for transmitting the information to a base unit; and 
of the tip is useful for identifying the local context of io (c) a base unit for receiving information generated by the 
the document an exact location of the PI pen tip by pen-instrument, interpreting the information and modi- 
recognizing pre-printed marks; or fy in g a stored electronic version of the physical docu- 

(c) a CCD array 240 at the back (i.e. "eraser") end of the * ent and displaying the interpreted results to the user 
PI because it is more intuitively acceptable than a side f° r acceptance or correction. 

, Therefore, the CB system description has served as an 

moune ^y- adequate vehicle for supporting a general manual entry 

Regardless of the location the CCD sensors must be able in(e ? active and el ^ trom > do * ument handli an J 

to read digitally encoded and pre-pnnted information and processing system. A specialized application program pack- 
support at least rudimentary optical character recognition 6 would customize thc system described for each distinct 
(OCR). The particular choice of CCD will dictate the application and document type. 

associated illumination requirements. 20 Although the previous description of the PI and its use 

A communication path 93 is needed for linking PI 91 with emphasized optical reading and message writing on a docu- 

base unit 92 where most signal processing of the PI sensor mcnt surface, it should be recognized that the PI may also 

occurs. A wired communication link 93 can use a small provide information from its gyroscopes and/or accelerom- 

tether connecting the PI to base unit 92 but a wireless eters for describing non-contact gestures unassociated with 

connection using either infrared or radio techniques would 25 physical writing on a document. Two cited examples of this 

be preferred. For this purpose transmitter/receiver unit 230 form of input to the system are the "check" signal indicating 

is incorporated in PI 91. affirmation (correct) and the "X" signal indicating negation 

FIG. 11 is a block diagram of the PI electronic system (wrong). Other gestures may be designed for use in a given 

showing physical inputs to the system at left that include context to indicate special messages (e.g., in editing, bori- 

forces associated with motion, pen-tip forces due to writing, 30 zontal oscillatory motions may indicate deletion if per- 

and reflected light for optical ready. These physical inputs formed over a previously written message or it may indicate 

are applied respectively to position sensor 210, pressure an "underline" if not performed over a written message but 

sensors 100, and optical sensors 220, 240, and 260. The under the message), 

sensor outputs are multiplexed into the modulator of trans- Selective use of color may also be used to distinguish 

mitter 230 by multiplexer (MUX) 275. MUX 275 may be 35 entries from authorized multiple users by generating an I.D. 

either a time-division or frequency multiplexer. Overall code for each user's PI at the outset of writing session. A 

control of the PI system is provided by controller 270. color display could readily identify the source of entries by 

B. 3.2 Base Unit 92 using a different color coding for each user. 

Base unit 92, shown in FIG. 12, communicates with PI 91, One or more buttons may be provided to initiate certain 

receiving sensor signals, processes the sensor signals, and 40 actions such as indicating that a bar code is to be read, 

drives any feedback device such as its display unit 95. The The PI can be used for other purposes than writing. For 

base unit is similar to a notebook keyboard computer example, the PI can function as a pointer or as a "master 

comprising a CPU 940, keyboard 943, memory 942, and mouse" that drives a slave mouse cursor on a graphic display 

display 95. In addition, the signal conditioning unit 96 or for navigating 3-dimensional virtual reality displays. A 

includes a transmitter/receiver as part of the communication 45 button can be used as a mouse button, 

channel interface 961 with communications link 93, and Yet another variation of the PI can have a receiver for 

demux 962, and signal conditioning electronics (963-966) receiving the output of multiple (infrared) transmitter for 

for preconditioning sensor signals from interface 961 and position location using triangulation techniques. This varia- 

analog-to-digital converters for feeding digitized sensor tion could be used for blackboard writing or other large 

related signals through mux 967 to the CPU for processing. 50 surface writing. and display boards. 

The base unit can also connect with Wide Area Networks In the foregoing specification, the invention has been 

through I/O interface 944 and is able to retrieve and modify described with reference to specific embodiments thereof. It 

electronic versions of the user's documents. An optional will, however, be evident that various modifications and 

auxiliary processor 941 (shown dotted) can be used to changes (as described above) may be made thereto without 

augment CPU 940 for executing written message recogni- 55 departing from the broader spirit and scope of the invention 

tion algorithms that require more intensive digital signal as set forth in the appended claims. The specification and 

processing than that usually done by the CPU. drawings are, accordingly, to be regarded in an illustrative 

Advances in micro-electronic techniques can allow most rather than a restrictive sense, 

of the functions of the base unit to be incorporated in the PI What is claimed is: 

unit, particularly the sensor signal conditioning, binary 60 1. A document handling system comprising: 

coded signal generation, and cache memory for buffering a pen instrument to track writing motions made with the 

and temporary storage. pen instrument so that an electronic document is modi- 

C. Other Applications and Variations fied by incorporating reproductions of strokes made 
The above description of a CB system that uses a pen- with the pen instrument, wherein the pen instrument 

instrument and base system was used as a convenient means 65 tracks its own motion and generates signals that rep- 

to describe the operation of a manual entry interactive paper resent the writing motions made with the pen instru- 

and electronic document handling and processing system. ment; and 
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a processing unit having an interface to receive the signals 
from the pen instrument, wherein the processing unit 
stores the electronic document and modifies the elec- 
tronic document in response to the strokes made with 
the pen instrument such that representations of the 
writing motions made with the pen instrument are 
displayed when the electronic document is displayed. 

2. The system of claim 1 wherein the pen instrument 
comprises at least one transducer to sense self motion in a 
multi-dimensional space and generate electrical signals rep- 
resentative of the self motion. 

3. The system of claim 1 wherein the pen instrument has 
at least one rate gyroscope to generate at least some of the 
electrical signals. 

4. The system of claim wherein the pen instrument has at 
least one accelerometer to generate at least some of the 
electrical signals. 

5. The system of claim 1 wherein the pen instrument has 
at least one accelerometer and at least one gyroscope to 
generate at least some of the electrical signals. 

6. The system of claim 1 further comprising at least two 
infrared receiving scanning stations located at prescribed 
positions external to the pen instrument, the scanning sta- 
tions to scan the area in which the pen instrument is used, 
wherein the scanning stations receive an infrared signal from 
the pen instrument. 

7. The system of claim 1 further comprising at least two 
infrared radiating scanning sources located at prescribed 
positions external to the pen instrument, the scanning 
sources to illuminate the pen instrument, wherein the pen 
instrument determines movements based, at least in part, on 
the scanning sources. 

8. A method comprising: 

tracking a location of a pen instrument when the pen 
instrument is used to modify a physical document, 
wherein sensors within the pen instrument track the 
location of the pen instrument; 

generating signals describing movement of the pen instru- 
ment based on the location of the pen instrument; 

transmitting the signals to a receiving device, wherein the 
signals are used to modify an electronic document 
corresponding to the physical document, and further 
wherein strokes made with the pen instrument are 
displayed when the electronic document is displayed. 

9. The method of claim 8 wherein generating electrical 
signals describing the series of strokes comprises generating 
the electrical signals with at least one transducer to sense self 
motion in a multi-dimensional space. 

10. The method of claim 8 wherein generating electrical 
signals describing the series of strokes comprises generating 
the electrical signals with at least one rate gyroscope. 

11. The method of claim 8 wherein generating electrical 
signals describing the series of strokes comprises generating 
the electrical signals with at least one accelerometer. 

12. The method of claim 8 wherein generating electrical 
signals describing the series of strokes comprises generating 
the electrical signals with at least one accelerometer and at 
least one gyroscope. 
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13. The method of claim 8 wherein tracking a location of 
a pen instrument comprises generating an infrared signal 
with the pen instrument that is received by at least two 
infrared receiving scanning stations located at prescribed 

5 positions external to the pen instrument, the scanning sta- 
tions to scan the area in which the pen instrument is used. 

14. The method of claim 8 wherein tracking a location of 
a pen instrument comprises receiving an infrared signal from 
at least two infrared radiating scanning sources located at 

10 prescribed positions external to the pen instrument, the 
scanning sources to illuminate the pen instrument, wherein 
the pen instrument determines movements based, at least in 
part, on the infrared signals. 
15 15. An apparatus comprising: 

means for tracking a location of a pen instrument when 
the pen instrument is used to modify a physical 
document, wherein sensors within the pen instrument 
track the location of the pen instrument; 
20 means for generating signals describing movement of the 
pen instrument based on the location of the pen instru- 
ment; 

means for transmitting the signals to a receiving device, 
25 wherein the signals are used to modify an electronic 
document corresponding to the physical document, and 
further wherein strokes made with the pen instrument 
are displayed when the electronic document is dis- 
played. 

3Q 16. The apparatus of claim 15 wherein the means for 
generating electrical signals describing the series of strokes 
comprises at least one transducer to sense self motion in a 
multi-dimensional space. 

17. The apparatus of claim wherein the means for gener- 
35 ating electrical signals describing the series of strokes com- 
prises at least one rate gyroscope. 

18. The apparatus of claim 15 wherein the means for 
generating electrical signals describing the series of strokes 
comprises at least one accelerometer. 

40 19. The apparatus of claim wherein the means for gener- 
ating electrical signals describing the series of strokes com- 
prises at least one accelerometer and at least one gyroscope. 

20. The apparatus of claim 15 wherein the means for 
tracking a location of a pen instrument comprises means for 

45 generating an infrared signal with the pen instrument that is 
received by at least two infrared receiving scanning stations 
located at prescribed positions external to the pen 
instrument, the scanning stations to scan the area in which 
the pen instrument is used. 

5Q 21. The apparatus of claim 15 wherein the means for 
tracking a location of a pen instrument comprises means for 
receiving an infrared signal from at least two infrared 
radiating scanning sources located at prescribed positions 
external to the pen instrument, the scanning sources to 

ss illuminate the pen instrument, wherein the pen instrument 
determines movements based, at least in part, on the infrared 
signals. 

***** 
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